Information processing apparatus

ABSTRACT

An information processing apparatus comprising first and second display units for respectively displaying first and a second composite images for the two eyes of a user, comprising: a moving unit configured to move positions of the first and second display units; a detecting unit configured to detect moving amounts of the first and second display units; first and second image capturing units configured to respectively obtain first and second captured images; an extracting unit configured to generate first and second extracted images by respectively extracting portions of the first and second captured images in extraction ranges associated with the moving amounts; and a composite image generating unit configured to generate the first and second composite images by respectively compositing first and second CG images with the first and second extracted images.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an information processing apparatusand, more particularly, to an information processing apparatus includingan image capturing system including at least two cameras for capturingan external image of the real world, and a display system forsuperposing a CG (Computer Graphics) image such as 3D-CAD data on theimage captured by the image capturing system, and displaying theobtained image.

2. Description of the Related Art

The MR (Mixed Reality) technique can exhibit an object drawn by CG orthe like as if the object existed in the real world. The MR technique isimplemented by superposing a CG image of a virtual world generated by CGon a captured image of the real world, and displaying a composite image.A video see-through type HMD (Head Mounted Display) is known as an imagedisplay apparatus for implementing the MR technique.

The video see-through type HMD includes an image capturing system forcapturing an external image of the real world, a CG image generator forgenerating a CG image of a virtual world as if the image were seen froman image capturing position in the real world, a composite imagegenerator for generating a composite image by compositing the externalimage and CG image, and a display system for displaying the compositeimage.

The image capturing system captures an external image of the real worldin a state in which the image capturing direction is almost matched withthe line of sight from the pupil position of the user, thereby obtaininga captured image. Note that in the image capturing system, right andleft cameras are arranged such that their optical axes are almostparallel. A human observes an object by rotating the right and left eyesinward so that the optical axes of the two eyes intersect at the object.This rotating motion of the eye is called convergence (congestion). Acaptured image obtained by the image capturing system has noconvergence, because the optical axes of the right and left cameras arealmost parallel. To obtain a natural image by giving convergence to thecaptured image having no convergence, a process of extracting a specificrange from the captured image is performed. This extraction process canprovide an image in which the two optical axes of the image capturingsystem virtually intersect at an object, so the user can feel as if heor she were observing an external world with convergence. The HMDprovides the user with a composite image obtained by superposing a CGimage on a captured image extracted to have convergence. The HMDmeasures the position and posture of the main body, superposes agenerated CG image on a captured image in accordance with the measuredposition and posture, and displays the composite image, therebydisplaying the captured image of the real world and the CG image of avirtual world in a correct positional relationship. To display thecaptured image and CG image in the correct positional relationship, itis necessary to measure the camera parameters such as the relativepositions, focal lengths, principal point positions, and distortions ofthe cameras of the image capturing system. In the image capturing systemof the HMD, the positional relationship between the two cameras is fixedin order to fix the camera parameters. The HMD displays a natural imagefor the user by matching the optical axis of the display system with theoptical axes of the two cameras of the image capturing system.

The pupil distances of humans have individual differences (in otherwords, humans do not all have the same distance between their pupils).Therefore, to reduce the influence of the individual difference of theHMD user on the impression of use, there is a method of adjusting thepupil distance of the HMD. When this pupil distance adjustment isperformed, the image capturing system and display system can be arrangedin positions matching the pupil distance of the user. This makes itpossible to provide the HMD for more users.

Japanese Patent Laid-Open No. 2005-311754 has disclosed a method ofmatching the image capturing system and display system with the pupildistance of the user by simultaneously moving the two systems.

In Japanese Patent Laid-Open No. 2005-311754, however, the cameraparameters such as the relative positional relationship between theright and left cameras change because the image capturing system movessimultaneously with the display system. This change in camera parameterscaused by the movement of the image capturing system sometimes makes itimpossible to correctly superpose a CG image on a captured image.

Also, convergence has individual differences because the pupil distancesof the users are different. In Japanese Patent Laid-Open No.2005-311754, the extraction range is not changed in accordance with theindividual difference of convergence, so as to give convergence to acaptured image. This sometimes makes it impossible to provide a capturedimage having convergence matching the user.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of the aboveproblems, and provides a technique capable of displaying a capturedimage having convergence matching the pupil distance of the user byperforming pupil distance adjustment without changing camera parameters.

According to one aspect of the present invention, there is provided aninformation processing apparatus comprising a first display unit and asecond display unit for respectively displaying a first composite imageand a second composite image for the two eyes of a user, comprising: amoving unit configured to move positions of the first display unit andthe second display unit; a detecting unit configured to detect movingamounts of the first display unit and the second display unit; a firstimage capturing unit and a second image capturing unit configured torespectively obtain a first captured image and a second captured image;an extracting unit configured to generate a first extracted image and asecond extracted image by respectively extracting portions of the firstcaptured image and the second captured image in extraction rangesassociated with the moving amounts; and a composite image generatingunit configured to generate the first composite image and the secondcomposite image by respectively compositing a first CG image and asecond CG image with the first extracted image and the second extractedimage.

Further features of the present invention will be apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a view showing an example of the arrangement of an informationprocessing apparatus;

FIG. 2 is a front view of an image display unit when the user isobserved from the object side of image capturing systems; and

FIG. 3 is a model view for explaining the extraction ranges of capturedimages.

DESCRIPTION OF THE EMBODIMENTS

An exemplary embodiment(s) of the present invention will now bedescribed in detail with reference to the drawings. It should be notedthat the relative arrangement of the components, the numericalexpressions and numerical values set forth in these embodiments do notlimit the scope of the present invention unless it is specificallystated otherwise.

First Embodiment

First, the overall arrangement of an information processing apparatusaccording to this embodiment will be explained below with reference toFIG. 1. This information processing apparatus includes an image displayunit 1, and image signal output units 2R and 2L for outputting imagesignals to the image display unit 1.

The image display unit 1 includes display systems 4R and 4L (first andsecond display units), a pupil distance adjuster 5, a pupil distancesignal detector 6, a pupil distance signal output unit 7, compositeimage input units 8R and 8L, image capturing systems 9R and 9L (firstand second image capturing units), and captured image output units 10Rand 10L.

The display systems 4R and 4L respectively include display elements 14Rand 14L and enlarging optical systems 15R and 15L, and display imagesfor the two eyes (a right eye 3R and left eye 3L) of the user. Each ofthe display elements 14R and 14L is a liquid crystal display or organicelectroluminescent (EL) display. The enlarging optical systems 15R and15L enlarge images displayed on the display elements 14R and 14L. Theimages enlarged by the enlarging optical systems 15R and 15L areprojected onto the right eye 3R and left eye 3L of the user. The pupildistance adjustor 5 can adjust the positions of the display systems 4Rand 4L in accordance with the pupil distance of the user.

The image capturing systems 9R and 9L respectively include imagecapturing elements 11R and 11L such as CCD sensors, and image capturinglenses 12R and 12L.

The image signal output units 2R and 2L respectively include capturedimage input units 13R and 13L, pupil distance signal input units 23R and23L, CG image generators 24R and 24L, image processors 25R and 25L, andcomposite image output units 26R and 26L.

The image capturing lenses 12R and 12L respectively form optical imagesof external images of the real world on the image capturing surfaces ofthe image capturing elements 11R and 11L. These optical images areconverted into electrical signals through photoelectric conversion bythe image capturing elements 11R and 11L, and the electrical signals areoutput as first and second captured images. The output signals from theimage capturing elements 11R and 11L are respectively input to thecaptured image input units 13R and 13L of the image signal output units2R and 2L via the captured image output units 10R and 10L.

The pupil distance signal detector 6 detects a pupil distance signalfrom the position information of the display elements 14R and 14Ladjusted by the pupil distance adjustor 5. The pupil distance signaloutput unit 7 outputs the pupil distance signal detected by the pupildistance signal detector 6 to the pupil distance signal input units 23Rand 23L.

The image processors 25R and 25L have a function of performing a processof extracting specific ranges of the first and second captured imagesobtained by the captured image input units 13R and 13L, and changing theextraction ranges based on the pupil distance signal (pupil distancedata) obtained by the pupil distance signal input units 23R and 23L.That is, the image processors 25R and 25L generate first and secondcomposite images by executing the image processing based on the CGimages generated by the CG image generators 24R and 24L, the first andsecond captured images obtained by the captured image input units 13Rand 13L, and the pupil distance signal (pupil distance data) obtained bythe pupil distance signal input units 23R and 23L. The composite imageoutput units 26R and 26L output the composite images generated by theimage processors 25R and 25L to the composite image input units 8R and8L. The composite images obtained by the composite image input units 8Rand 8L are displayed on the display elements 14R and 14L. The enlargingoptical systems 15R and 15L enlarge the images displayed on the displayelements 14R and 14L, and project the enlarged images onto the right eye3R and left eye 3L of the user.

Next, a front view of the image display unit 1 when the user is observedfrom the object side of the image capturing systems will be explainedwith reference to FIG. 2.

The image capturing systems 9R and 9L of the image display unit 1 areintegrally fixed by a fixing unit 16. The fixing unit 16 is formedindependently of the display systems 4R and 4L. The image display unit 1includes a feed screw 17 (a screw mechanism) and two guide bars 18 and19 (guide members) as a moving mechanism of the display systems 4R and4L.

Threaded portions 20R and 20L are formed on those portions of the feedscrew 17, which are threadably engaged with (fitted by a screw actionin) the display systems 4R and 4L. The screw portions 20R and 20L areformed in opposite directions, and have a reverse screw relationship.Also, a manual adjustment means, e.g. an adjusting knob 21, is connectedto the feed screw 17. When the user rotates the adjusting knob 21, thedisplay systems 4R and 4L can move nearer to or away from each other asindicated by arrows B and C through the screw mechanism along the guidebars 18 and 19.

The two guide bars 18 and 19 prevent the display systems 4R and 4L fromrotating relative to each other, and allow them to accurately move inthe directions of the arrows B and C. This moving mechanism enables theuser to adjust the display systems 4L and 4R by moving them to positionsfacilitating easy observation (of) the user.

Furthermore, the feed screw 17 includes the pupil distance (signal)detector 6. The pupil distance detector 6 includes a rotary encoder 22.The rotary encoder 22 detects the rotational angle of the feed screw 17,and outputs the detection result to the pupil distance signal outputunit 7.

The pupil distance signal output unit 7 has a memory. A data tableindicating the relationship between the rotational angle of the feedscrew 17 and the pupil distance is stored in the memory. The pupildistance signal output unit 7 reads out, from the data table in thememory, pupil distance data corresponding to the detection result fromthe rotary encoder 22, and outputs the pupil distance data to the pupildistance signal input units 23R and 23L.

The ranges of captured image extraction executed by the image processors25R and 25L by using the pupil distance data obtained by the pupildistance signal input units 23R and 23L will be explained below withreference to FIG. 3.

First, the optical axes of the image capturing systems 9R and 9L arealmost parallel. Captured images 27R and 27L indicate images captured bythe image capturing systems 9R and 9L having almost parallel opticalaxes. First and second extracted images 28R and 28L indicate capturedimages formed by extracting partial ranges of the first and secondcaptured images 27R and 27L. First and second captured images 29R and29L indicate captured images formed by changing the extraction rangesfrom those of the extracted images 28R and 28L, so that the optical axesof the image capturing systems virtually have specific convergence.

The extraction ranges of the extracted images 29R and 29L havingconvergence are determined based on the pupil distance data obtained bythe pupil distance signal input units 23R and 23L. This makes itpossible to provide captured images having convergence matching theuser.

Assume that the extraction ranges of the extracted images 29R and 29Lhaving convergence are changed to positions where the optical axes ofthe image capturing systems 9R and 9L intersect at points moved by ypixels in the directions of the arrows B. A moving amount y of theextraction range is represented by

$\begin{matrix}{y = \frac{\left( \frac{D + X}{2} \right)A}{2L\; {\tan \left( \frac{\omega}{2} \right)}}} & (1)\end{matrix}$

where y: the moving amount (pixels) of the extraction range, D: thepupil distance (mm) of the image capturing systems, X: the moving amount(mm) of the display system, A: the number of horizontal pixels (pixels)of the image capturing system, L: a distance (mm) to the convergencepoint of the image capturing system, and w: the angle of view (degrees)of the image capturing system.

Note that the moving amount X of the display system takes a negativevalue when the display system is moved in the directions of the arrowsB, and a positive value when the display system is moved in thedirections of the arrows C.

By changing the extraction ranges in accordance with equation (1), it ispossible to give captured images convergence matching the pupil distanceof the display systems, and provide a natural comfortable image for theuser.

The CG image generators 24R and 24L generate first and second CG imageshaving a parallax, based on the position-posture information of theimage display unit 1. The image processors 25R and 25L superpose thefirst and second CG images generated by the CG image generators 24R and24L on the captured images extracted in accordance with equation (1),thereby executing a composite image generating process.

As described above, it is possible to correctly superpose CG images oncaptured images by fixing the camera parameters, and display capturedimages having convergence matching the pupil distance of the user.

Note that the method of performing pupil distance adjustment topositions appropriate for the user depends on the user's judgment inthis embodiment, but it is also possible to capture images of the pupilsof the user, measure the pupil distance of the user from the capturedpupil images, and perform pupil distance adjustment to appropriatepositions.

Note also that the image capturing systems are fixed in order to fix thecamera parameters, and the feed screw having the right and left reversescrews is used as the mechanism for moving the display systems. However,another mechanism may also be used as long as the positions of thedisplay systems can be matched with the pupil distance of the user.

In this embodiment as has been explained above, pupil distanceadjustment is performed on the display systems while the image capturingsystems of the image display unit are fixed. To reduce the uncomfortablefeeling of the user resulting from the fixation of the image capturingsystems, the extraction ranges of the captured images are changed toranges associated with the moving amounts of the display systems.

In this embodiment, the distance between the image capturing systems isfixed, and the distance between the display systems is adjusted bymoving only the display systems. Consequently, pupil distance adjustmentcan be performed while the camera parameters are fixed, that is, withoutchanging them. Also, captured images having convergence matching theuser can be displayed by changing the extraction ranges of the capturedimages in accordance with the moving amounts of the display systems.This makes it possible to provide a natural image for the user.

In the present invention, it is possible to perform pupil distanceadjustment without changing the camera parameters, and display capturedimages having convergence matching the pupil distance of the user.

Other Embodiments

Aspects of the present invention can also be realized by a computer of asystem or apparatus (or devices such as a CPU or MPU) that reads out andexecutes a program recorded on a memory device to perform the functionsof the above-described embodiment(s), and by a method, the steps ofwhich are performed by a computer of a system or apparatus by, forexample, reading out and executing a program recorded on a memory deviceto perform the functions of the above-described embodiment(s). For thispurpose, the program is provided to the computer for example via anetwork or from a recording medium of various types serving as thememory device (for example, computer-readable storage medium).

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2011-282264 filed on Dec. 22 2011, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An information processing apparatus comprising afirst display unit and a second display unit for respectively displayinga first composite image and a second composite image for the two eyes ofa user, comprising: a moving unit configured to move positions of saidfirst display unit and said second display unit; a detecting unitconfigured to detect moving amounts of said first display unit and saidsecond display unit; a first image capturing unit and a second imagecapturing unit configured to respectively obtain a first captured imageand a second captured image; an extracting unit configured to generate afirst extracted image and a second extracted image by respectivelyextracting portions of the first captured image and the second capturedimage in extraction ranges associated with the moving amounts; and acomposite image generating unit configured to generate the firstcomposite image and the second composite image by respectivelycompositing a first CG image and a second CG image with the firstextracted image and the second extracted image.
 2. The apparatusaccording to claim 1, wherein said moving unit comprises: at least twoguide members; a screw mechanism formed along one of said guide members;and a rotatable manual adjustment unit connected to said screwmechanism, and said screw mechanism moves the positions of said firstdisplay unit and said second display unit along said guide members, inaccordance with the rotation of said rotatable manual adjustment unit.3. The apparatus according to claim 2, wherein those portions of saidscrew mechanism, which correspond to said first display unit and saidsecond display unit are threaded in opposite directions, and said movingunit moves said first display unit and said second display unit awayfrom or nearer to each other.